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KACZMARZ ALGORITHM IN HILBERT SPACE AND 

TIGHT FRAMES 

RYSZARD SZWARC 


Abstract. We prove that any tight frame in Hilbert space can 
be obtained by the Kaczmarz algorithm. An explicit way of con¬ 
structing this correspondence is given. The uniqueness of the cor¬ 
respondence is determined. 


1. Introduction 

Let {e„}^Q be a linearly dense sequence of unit vectors in a Hilbert 
space Ti. Define 

Xq = {x, eo)eo, 

The formula is called the Kaczmarz algorithm ([ 21 )- 

It can be shown that if vectors the Qn are given by the recurrence 
relation 

n—1 

(1) 90 t'O) 9n Cn ^ Qi 

i=0 

then Qq is orthogonal to Qn-, for any n > 1 and 

n 

(2) Xn = ^{x,gi)ei. 

i=0 

By dD) the vectors {gn}’^=o linearly dense in H. Also by definition 
of the algorithm the vectors x — Xn and e„ are orthogonal to each other. 
Hence 

||x|p = \\x - XolP + \{x,go)\^, 

(3) \\x - Xn-l\\‘^ = \\x - XnW'^ + \{x,gn)\‘^, U > 1. 
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For n > 1 let S'„ denote the finite dimensional operator defined by 
the rule 

n 

(4) SnV = '^{y, ej)gj, y eU. 

j=0 

Observe that the formulas CD and m can be restated as 

(5) (h Qji 

(6) (/ — S*)x = X — Xn- 

Moreover by m it follows that 

n 

(7) ||x-x„|p = ||(/-5;;)x|p = \\xf -'^\{x,gj)\‘^. 

j=0 

In particular 

CX) 

( 8 ) '^\{x,gn)\‘^ <\\xf, xen. 

n=0 

The sequence is called effective if x„ —*• x for any x G 7i. 

By virtue of CD this is equivalent to ||x|p = for any 

X G 7i, which means i® a normalized tight frame. We refer to 

|3] and jl] for more information on Kaczmarz algorithm. 

Acknowledgement. I thank Wojtek Czaja and Pascu Gavruta for 
pointing my attention to Lemma 3.5.1 of p]. 

2. Bessel sequences 

Definition 1. A sequence of vectors in a Hilbert space H will 

be called a Bessel sequence if holds. The sequence {gn}’^=o will be 
called a normalized Bessel sequence if in addition Unroll = 1- 

Let Pn denote the orthogonal projection onto ef; the orthogonal com¬ 
plement to the vector e„. By 0(1)] we have 

(9) I-S: = PnPn-l...Po, 

(10) I — Sn = Pq ■ ■ ■ Pn-lPn- 

Theorem 1. For any normalized Bessel sequence {gn}1^=o i'n o Hilbert 
space TL there exists a sequence {e„}^o r*/ unit vectors such that m 
holds. In other words any normalized Bessel sequence can be obtained 
through Kaczmarz algorithm. 
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Proof. We will construct the sequence {en}'^=o recursively. Set cq = go. 
Assume the unit vectors ei,..., ctv-i have been constructed such that 
the formula m holds for n = 0,..., — 1. We want to solve in y the 

equations 

(11) {I - SN-i)y = gN, Ibll = 1- 

Bydini) we have {I — SN-i)e-N-i = 0, i.e. the operator I — Sn-i admits 
nontrivial kernel. Hence the solvability of m is equivalent to that of 

(12) {I - SN-i)y = gN, Ibll < 1- 

By the Fredholm alternative the equation (/ —Fat-i)?/ = gN is solvable 
if and only if gjsf is orthogonal to keT{I — We will check that this 

condition holds. Let x G ker(J — *S'^_i). Then by ((7j) and (jHI) we have 

A^—1 oo 

0 = ||(/- = ||xf - 

j=0 j=N 

In particular {x,gN) = 0, i.e. gN T ker(/ — 

Let y denote the unique solution to 

{I - SN-i)y = gN, y ±keT{I - Sn_i). 

The proof will be complete if we show that ||i/|| < 1. Again by the 
Fredholm alternative we have y G Im (/ — *S'j^_i). Let y = {I — S'^_i)x 
for some x eH. We may assume that x T ker(/ — In particular 

{x,go) = 0, as Q yields go G ker(/ - By dTj) we have 

N-l 

Halt = ll(/-^ 1(1,»)L 

i=i 

One the other hand 

\\yf = {x, {I - SN-i)y) = {x, gN)- 

Therefore 

N 

\\yf - \\yf = bir - ^ O’ 

i=i 

which implies ||i/|| < 1. □ 

Corollary 1. For any normalized tight frame {gn}’i^=o ® Hilbert 

space H there exists an effective sequence {cnflfLo vectors such 

that m holds, i.e. any normalized tight frame can be obtained through 
Kaczmarz algorithm. 


4 


RYSZARD SZWARC 


For a sequence {e^j^Q of unit vectors the normalized Bessel se¬ 
quence {gn}’^=o is determined uniquely. However a given normalized 
Bessel sequence may correspond to many sequences of unit vectors due 
to two reasons. First of all for certain N the dimension of the space 
ker(J — S'at-i) may exceed 1. Secondly, if we £x a unit vector u in 
ker(J — Sn-i) the vector can be dehned as cat = y -|- An for any 
complex number such that |Ap-|-||y|p = l. In what follows we will in¬ 
dicate properties which guarantee one to one correspondence between 
{en]n=Q and {gn]n=Q- 

Definition 2. A sequence of unit vectors will be called stable if 

the vectors {(^n}n=N are linearly dense for any N. A normalized Bessel 
sequence {gn}’^=o will be called stable if the vectors {(/o} U {gn}'^=N 
linearly dense for any N. 

Proposition 1. Let sequences {e^l^Q and {gn\^=o satisfy (Op. The 
sequence stable if and only is stable and (e„, e„+i) ^ 

0 for any n > 0. 

Proof. Assume {gn}'^=o is stable. First we will show that the kernel 
of / — Sn -1 is one dimensional and thus consists of the multiples of 
the vector cn-i (see (HDD. Assume for a contradiction that dimker(/ — 
Sn-i) > 2. By the Fredholm alternative we get dimker(/— > 2. 

Hence there exists a nonzero vector x such that x -L go and (/ — 
Sfi_i)x = 0. By 0 we obtain 


N-l 

n=l 

This and the condition (jH)) imply that x is orthogonal to all the vectors 
go and {gn}’^=Ny which contradicts the stability assumption. 

Assume {cn-i^cn) = 0 for some A^ > 1. Then by (fTn|) we have 
cat-i, Cat G ker(/ — Fat-i) which is a contradiction as the kernel is one 
dimensional. 

Concerning stability of {e„}^Q assume a vector y is orthogonal to all 
the vectors In particular y is orthogonal to Cat. Since ker(/ — 

Sn) = Ccat by the Fredholm alternative y belongs to Im (/ — Sfj). 
Let y = {I — Sff)x for some x E Ti. We may assume that x -L go as 
go E ker(J- S*^). By 0, since y is orthogonal to e„ for n > N, we get 
y = {I — S*)x = (/ — Sff)x for n > N. On the other hand by 0 and 
0 we obtain that (x, gn) = 0 for n > A^ -f 1. Since x T 5^0 by stability 
assumptions we obtain x = 0 and thus y = 0. 
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For the converse implication assnme is stable and (e^, e^+i) 7 ^ 

0. By the inequality (see 0) 

\\x-Xn\\ > |(e„_i, e„)|||x - x„_i|| 

we get that a: — 7 ^ 0 for any x _L eg. Since x — Xn = {I — S*)x the 

kernel of / — S'* consists only of the multiples of co = go- 

Let X be orthogonal to {< 70 } U {gn}n>N+i for some > 1. By 0 
we obtain that Xn = xjsi for n > N. By the dehnition of the Kaczmarz 
algorithm we get x — xat _L e„ for n > + 1. Now stability of {e„}(^Q 

implies x — xat = 0. By (jH)) we obtain (/ — S^)x = 0. This implies 
X = 0 since the kernel is one dimensional and consists of the multiples 
of go- □ 

For sequences {cnjj^g and {cr„e„}^Q, where cr^ are complex numbers 
of absolute value 1, the Kaczmarz algorithm coincide. Therefore we will 
restrict our attention to admissible sequences of unit vectors {e„}^o 
such that {e„,e„+i) > 0 . 

Theorem 2. Let {gn}'^=o be a stable normalized Bessel sequence. Then 
there exists a unique admissible sequence {e„}(^g of unit vectors such 
that m holds. Moreover the sequence {e„}(^Q is stable. 

Proof. The proof will go by induction. The vector eg is determined by 
eg = go- Assume the vectors eg,...,eAr_i were determined uniquely. 
We have to show that the problem 

{I - SN-i)y = gN, Ibll = 1, {y,eN-i)>0 

has the unique solution y. 

By the proof of Proposition 2 the kernel of I—Sn-i is one dimensional 
and thus consists of the multiples of the vector cat-i. By the proof of 
Theorem 1 there exists the unique solution yi >4 to the problem 

(/ - SN_i)y = gN, 27 -L ker(/ - Sn-i) 

and IIt/atII < 1. Moreover by this proof ||27vil = 1 if and only if 

N 

= 0 , 

7=1 

where y^ = {I — Sf;_i)x and x T ker(/ — This leads to a 

contradiction because by inequality (0 we get that x is orthogonal to 
all the vectors go and Hence i| 2 /Ar|| < 1. 

At this stage we know that any solution to the equation 

(/ - SM-i)y = 0 
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is of the form 

y = Vn + Aejv_i, A G C 

because ker(J — S'at-i) = Ccat-i. Since ||i/Ar|| < 1 and yjq ± eAr_i there 
exists a unique solution y satisfying ||y|| = 1 and (?/, cat-i) > 0 namely 
the one corresponding to A = □ 

Corollary 2. Let {gn}^=o be a stable normalized tight frame. Then 
there exists a unique admissible effective sequence {e„}^Q of unit vec¬ 
tors such that m holds. Moreover the sequence is stable. 

3. Algorithm 

The proof of Theorem 1 can also be given by using Gram matrix of 
the sequence This argument can be used for constructing an 

underlying sequence of unit vectors {e„}^Q. This will be done below. 

The following lemma follows from Lemma 1 it follows from ^ Lemma 
3.5.1], 

Lemma 1. The collection {gnfffLQ is a Bessel sequence if and only 
if the Gram matrix G = {{gi, gj)}ffj=o corresponds to a contraction 
operator on The sequence {gn}fLo is a tight frame if and only if 

it is linearly dense and G corresponds to a projection on 

Let {cnffLQ be a sequence of unit vectors in a Hilbert space Ti and 
let {gn}ff=o be the corresponding normalized Bessel sequence. Let M 
be the strictly lower triangular part of the Gram matrix of the sequence 
{cnffLo U strictly lower triangular matrix dehned by 

{I + U){I + M) = I. 

By Q the matrix G is a contraction on the Hilbert space 
Lemma 2. For any i,j we have 

{g„gf = {{I-UU*)5j,S,)p(^n) 

Proof. Let 


By g 
(13) 



ruio 0 0 0 0 ... 


cio 0 0 0 0... 

M = 

^20 m2i 0 0 0 ... 

, U = 

C20 C21 0 0 0 ... 


mgo mgi 77132 0 0 ... 


C30 C31 C32 0 0 ... 


^ ; ; ; ; ■■.) 


^ ; ; ; ; 


we have 


i—1 

Qi 62 ^ ^ 


k=0 
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Set Cnn = fTinn = 1 and let 1 < fc < n. By taking inner prodnct with gj 
in (HI we get 


i j 

{Qii 9j) ^ ^ Cjfc ^ ^ Cl) 

k=0 1=0 

= {{I + U){I + M + M*){I + U*)6^, 6,)piny 
Taking into account relations between the matrices M and U gives 


(14) {I + U){I + M + M*){I + U*) = I - UU\ 


The product of these matrices is well dehned since U* leaves the space 
F(N) = span{(5„ | n > 0}) invariant. Therefore 




□ 


Remark. Lemma El can be used to give a shorter and simpler proof 
of Theorem 1 in j3] . Indeed, by Lemma ^ a linearly dense sequence 
{9n\’^=o constitutes a normalized tight frame if and only if the matrix 
G = {(fi'i, 5'i)}^=o is ^ projection. In view of Lemma El the latter is 
equivalent to U being a partial isometry. Moreover in this case we have 

OO 

dimLf = 5^||^„||2 = Tr(J-f/f/*). 

n=0 

We are ready now to give an alternative proof of Theorem 1. Let 
be a normalized Bessel sequence. By Lemma ^ the matrix 
A\= I — G is positive dehnite. Moreover ^4(0, i) = A{i, 0) = 0 because 
ll^foll = 1 ciLLcl go T gi for i > 1. Let A denote the truncated matrix 
obtained from A by removing the hrst row and the hrst column. Clearly 
A corresponds to a positive dehnite contraction on The next 

lemma is probably known and provides inhnite dimensional version of 
the so called Cholesky decomposition of positive dehnite matrices. 

Lemma 3. For any positive definite matrix B = {b{i, j)}ij=i there 
exists a lower triangular matrix V = {vli, j)}^=i such that B = VV*. 

Proof. By the well known fact there exists a Hilbert space Ai and a 
linearly dense sequence of vectors in At such that 

Khj) = {hi,hj). 

By applying the Gram-Schmid procedure to this sequence we obtain 
an orthonormal sequence where N = dimAI, such that hi G 
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span {rji, 772 , • • •, Vi} i + 1. In particnlar there are coefficients 
Vik for i > k and + 1 > A; for which we have 

i 

hi ^ ^ '^ikVk ■ 
k=l 

Set Vik = 0 for z > /c and for k > N. Then 

min(ij') 

b{hj) = {hi,hj)= ^ VikVjk = 
k,l=l 

□ 


By Lemma El there is a lower triangular matrix V = {vij}fj^i such 

that A = VV*. Let U = f^e strictly lower triangular 

matrix obtained from V by adding a zero row and a zero column, i.e. 


Cij — 


0 


ij > 0, 
ij = 0. 


In this way we obtain 


(15) I -G = A = UU*. 

Since G corresponds to a contraction on so does U. Let M = 

{'^ij}Tj=Q tie the strictly lower triangular matrix determined by (/ + 
M){I + U) = I. Set mu = 1 and dehne (cf. (P)) 


e* ^ ^ 'iii^ikQk • 
k=0 

We claim that e* are unit vectors and moreover (e,, Cj) = mij for i > j. 
This will give (P and thus conclude another proof of Theorem 1. By 
(uni) we have 


* ] 


I 3 


(e..e,> = EE T^ikT^jliVki 9l) EE mikmji{I - UU*){k, 1) 

k =0 1=0 k =0 1=0 

= {I + M){I -UU*){I + 

On the other hand (HI yields 

(/ + M){I - UU*){I + M*) = I + M + M*. 

In particular for i > j we obtain 


(e,, Cj) — 


rriij i > j, 


i = J. 
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This way of proving Theorem 1 provides an algorithm for construct¬ 
ing a sequence of unit vectors {e„}5^Q for a given normalized Bessel 
sequence {gn}’^=o- Indeed, it suffices to determine an algorithm for 
proving Lemma 121 When B is strictly positive definite then the solu¬ 
tion can be given by the so called Cholesky algorithm. When B is not 
necessarily strictly positive definite this algorithm fails and w ehave to 
find a different way of constructing the decomposition. 

We will construct a sequence of indices {nk}^=i in the following way. 
Let rii be the smallest number i such that > 0. If such number does 
not exist then B = 0. Assume ni, n 2 ,..., nfc have beeu coustructed iu 
such a way that the determinaut 

Afc = det{bninj)ij=i > 0. 

Theu let n^+i be the smallest number such that 

det{bn.nj)ij=i > 0. 

If such number does not exist the procedure terminates and N = rik. 

The matrix B gives rise to a positive definite hermitian form on the 
space F{N+) = span | n > 1} by the rule 

CXD 

{x,y) = b{ij)xiyj. 

i,j=^ 


Lemma 4. For any n there exist i with Hi < n < rij+i and numbers 
^nk for 1 < k < i, such that 


(16) 


^nkdru .) dn 


k=l 


0, m >1. 


Proof. If n = rij for some i, then statement follows as dn = dm- 
Otherwise we have Ui < n < Uj+i for some i. By plugging in m = 
77-1, ^ 2 ,..., Uj to (fTT)|l we obtain a system of linear equations 


^ ^ d^nkb 

k=l 


rik-ni 


b 


nni •> 


/ = l,2,...,z. 


The main determinant of this system is Aj. Therefore the system has 
the unique solution ..., \n,i. By definition of rij+i we have 



bnin2 

bnirii 

^nin 

bn2n\ 

bn2n2 

bn^rii 

bn2n 


bnin2 

dmui 


^nni 

bnn2 

b-nrii 

bnn 
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As Aj > 0 the first i rows of this matrix are linearly independent. 
Therefore the last row of this matrix is a linear combination of the 
hrst i. The coefficients mnst coincide with In particnlar 

considering the last entry of the rows gives 


J2Xnkbn,n = bn 


k=l 


This is eqnivalent to (HSl) with m = n. 

Since (HSl) is valid for m = n, ni,..., n* then 

i i 

I ^ ^ ^nkbn/f .) bn ^ ^ ^nkbn/c ) 0- 


k=l 


k=l 


By Schwarz ineqnality this implies for any m. 
Dehne the seqnence of vectors by the formnla 


- 1 A - 1 

^1 r-T— 


v/AT " v/A-^A- 

It can be checked easily that 

(17) {r]i,r]j) = 5l 

Obviously from the dehnition we have 

Vi / i bn^ ^ ^ Olikbv 
k=l 

for some explicitly given coefficients ctjfc. Therefore 


bnini 

bnin2 

bniUi 

bn2ni 

bn^n^ 

bn2ni 

bui-ini 

bni-in2 

bm-irii 

bni 

bn2 

bui 


i—1 


■"rik 


□ 


i 

(18) bm = ^ ^ PikVkj 

k=l 

for some coefficients j3ik- By (uni) and m we get that for any n there 
exist i with Ui <n < Uj+i and numbers Vnk ior 1 < k < i, such that 


Setting Vnk 
(19) 


bn ^ ^ 'iinkVk ; bn 
k=l 

0 for i < k < n gives 


bn ^ ^ 'iinkVk ; bn 


k=l 


0, m >1. 


0, m >1. 
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Therefore by (HED and (unD we have 


0 




Therefore B = VV* where 


{Sn,Sm) 


min(n,m) 

k=Q 


= b 


nm 


min(n,m) 

^ ^ '^nk'^mk- 

k=0 


/nil 0 

0 

0 

0 

•A 

T21 V22 

0 

0 

0 


T31 T32 

T33 

0 

0 


T41 V42 

T43 T44 

0 


\ ■■ 




•7 


By analyzing the entire constrnction we may conclnde the coefficients 
Vnk can be compnted in an algorithmic way. 


4. Equivalent sequences 

Any seqnence of nnit vectors {e^j^o leads by Kaczmarz algorithm 
to a normalized Bessel seqnence {gn}'^=Q- 

Definition 3. Two normalized Bessel sequences {gn}'^=o cind 
will be called equivalent if there is a unitary operator V such that g'^ = 
y9n for n > 0. Similarly two sequences and of unit 

vectors will be called equivalent if there is a unitary operator V such 
that e'^ = Ven for n> 0. 

It is easy to see that if the seqnences {e„}^Q and are eqniva- 

lent so are the corresponding seqnences of normalized Bessel seqnences 
{9n}'n=oy with the same nnitary operator V. The converse 
is not trne, as the normalized Bessel seqnences do not correspond to se¬ 
qnences of nnit vectors in one-to-one fashion. Nevertheless by Lemma 
1, the eqnivalence relation between normalized Bessel seqnences can be 
described in terms of Gram matrices of the corresponding seqnences of 
nnit vectors. 

Assnme seqnences of nnit vectors {e„}^Q and are associated 

with normalized Bessel seqnences {gn}’^f=o and {e^jj^g, respectively. 
Let M and M' be the strictly lower triangnlar part of the Gram ma¬ 
trices of seqnences {e„}^g and {e^j^g, respectively. Let U and U' be 
strictly lower triangnlar matrices defined by 

(/ + G)(/ + M) = (J + U'){I + M') = I. 
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By i the matrices U and U' are contractions on the Hilbert space 

Corollary 3. The sequences {9'n}^=o equivalent if and 

only ifUU* = U'U'*. 

Proof. By Lemma 1 we get that UU* = U'U'* if and only if {gi^gf) = 
{g'i,g'j) for any i,j > 0. Obviously the latter, along with the linear 
density of vectors {gn}1^=o and equivalent to the existence 

a unitary operator U such that g'^ = Ugi. □ 

Remark. It would be of interest to determine when two sequences 
of unit vectors and l^ad to the same normalized Bessel 

sequence {gn}n=o- 
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